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PREFACE 

This draft document consists of technical working material that has not been 
formally reviewed. It has b'.en prepared In this manner to provide timely 
documentation to personnel supportlr* the Foreign Commodity Production Fore- 
casting (FCPF) project and to provide others In the technical community with 
a mean*. fo»* staying abreast of FCPF project tasks. 

A high priority goal In the Agriculture and Resources Inventory Surveys Through 
Aerospice Remote Sensing (AgRISTARS) FCPF project Is the development of Landsat 
dependent-area estimation technologies to support at-harvest foreign corn and 
soybean production forecasting. The first large-scale test and evaluation 
of these technologies has been scheduled for Fiscal Year (FY) 1981/1982 In a 
pilot experiment In the U.S. Central Corn Belt (Iowa, Illinois, and Indiana). 
The basic objectives of the U.S. Pilot experiment have been outlined In the 
January FCPF Project Implementation Plan. Results of the expected evaluations 
will support adaptation of com and soybean area estimation technologies for 
Argentina and Brazil and provide Insight into the experiment design Issues 
for the Argentina Exploratory Experiment scheduled for FY1983. 

The major area estimation technology scheduled for evaluation In the pilot 
experiment Is tne U.S. Baseline Com and Soybean Segment Classification 
Procedure developed and delivered to the FCPF Project by the Environmental 
Research Institute of Michigan and the Space Sciences Laboratory of the 
University of California at Berkeley. The baseline procedure, derived from 
previous experiences for small grains and exploratory corn and soybeans pro- 
cedures, generates stratified areal estimates from analyst-labeled field- 
like targets. 

Prior to the Preliminary Design Review (PDR), budgetary and data procurement 
considerations resulted 1r definition of the following generalized experiment 
schedule: Intensive evaluation of technologies on 1978/1979 crop-year data 
beginning In FY1981 followed by a regional test o* the Integrated component 
technologies on 1980 crop-year data. The accompanying draft documentation 
represents the Preliminary Experiment Design and Plan for the 1978/1979 



crop -year data evaluations as presented to National Aeronautics and Space 
Administration (NASA) FCPF project management. Review comments were submitted 
to the Experiments Manager as written requests or recommendations for experi- 
ment plan and/or design changes (RID's). The RID's were to be reviewed and 
recommendations provided to NASA FCPF project management on their proper 
disposition prior to a critical design review of the final Experiment Plan. A 
summary of these RID's Is attached. 

However, revised budgetary considerations and changes 1r project objectives 
Immediately subsequent to the POR necessitated the U.S. Pilot Experiment be 
redesigned. As a result, modified versions of the Pilot Experiment Plans pre- 
sented herein for the 1978/1979 crop-year data evaluations will be synthesized 
with the 1980 crop-year data evaluations In a FY1981/1982 U.S. Corn/Soybean 
Pilot Experiment Plan. This new plan Is currently scheduled to be completed 
and presented to NASA FCPF project management In June 1981. 
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TEST AND EVALUATE TECHNOLOGY FOR GENERATING AT-HARVEST COfir- AND SOYBEANS AREA., YIELD, 
AND PRODUCTION ESTIMATES IN THE U.S. CENTRAL CORN BELT (iOWA, ILLINOIS, AND INDIANA). 
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FY1981 FCPF PROJECT U.S. CORN/SOYBEAN PILOT EXPERIMENT 
PLAN TO SUPPORT ARGENTINA AND BRAZIL CORH/SOYBEAN EXPERIMENTS 
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DEVELOPMENTAL AGRICULTURE REMOTE SEHSIHG INFORMATION SYSTEM (DARSIS) 





Crop Growth Stage Models for Corn and Soybeans 
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MAJOR SUBCOMPONENTS AND FUNCTIONAL FLOW 
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AVERAGE FARM SIZE WAS REPRESENTED IN TWO GROUPS: 
SMALL <£ 190 ACRES 
LARGE ± 190 ACRES 
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SUMMARY OF THE NUMBER OF AVAILABLE SEGMENTS 
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1978 tWLTICROP DATA SE T 

FOR WHICH GROUiiD-TRUTH TAPES (UGTT's) ARE AVAILABLE. 


ORIGINAL PAGE IS 
OF POOR QUALITY, 



9 • 

9 9 


IP 


a 


(Vi 


3> 




> 

u 

z 

LU 

3 

O 

LU 

OS 

U. 


13 


MIDPOINT OF THE 
NUMBER OF ACQUISITIONS 



ORIGINAL PAGE IS 
OF POOR QUALITY 



Previous *Iulticrop Data Analyses 
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QUANTITATIVE EVIDENCE THAT PROCEDURES MUST DEAL WITH BOUNDARY/MIXED PIXELS 



PROPORTION ESTIMATION 

LARGEST SOURCE OF ERRORS WAS IDENTIFIED AS DOT LABELING ERRORS 
- ABSOLUTE ERROR CORN -AS 
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CROP GROUP STRATA 
SPECTRAL CLUSTER (B-CLUSTER) 



COMPARISON OF ATTRIBUTES 
BETWEEN PROCEDURES 


PROCEDURE M 
FOR SPRING 
GRAINS 

HAZE 

SUN ANGLE 
SENSOR 

TASCAP G, B 

BLOB 

BCLUSTER 

NONE 

MIOZUNO 
40-100 BLOBS 

INTEGRATED 

PROCEDURE 1 

SUN ANGLE 
SENSOR 

TASCAP G, B 

DOT 

ISOCLAS 

40 DOT 
LABELS 

SYSTEMATIC- 
RANDOM 
60 DOTS 

HIERARCHICAL 

DECISION 

LOGIC 

BASELINE C/S 
PROCEDURE 

HAZE 

SUN ANGLE 
SENSOR 

TASCAP B 

GRABS 

TPC 

8018 

CROP GROUP 
STRATA 
B CLUSTER 

NONE 

MIDZUNO 
40-100 BLOBS 

‘CONVERGENCE 
OF EVIDENCE' 
DECISION 
LOGIC 

ATTRIBUTE 

PREPROCESSING 

FEATURE 

EXTRACTION 

TARGET TYPE 

STRATIFICATION 

METHOD 

z 

o 

IS) o 
•— Ul 

> at 
at — 

*3 

SB 

SAMPLING 

METHOD 

LABELING 

METHOD 
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(CONT.) 


ATTRIBUTE 

BASELINE C/S 
PROCEDURE 

PROCEDURE 1 

PROCEDURE M 
FOR SPRING GRAINS 

ESTIMATION 

METHOO 

STRATIFIED 

AREA1 

ESTIMATE 

STRATIFIED 

AREA1 

ESTIMATE 

STRATIFIED 

AREA1 

ESTIMATE 

ANALYST/ 

MACHINE 

INTERACTION 

INTEGRATED 

DISJOINT 

DISJOINT 
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ERIM STUDY IF TWO CROP SAMPLING WITH PR!ORS 
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FY1S81 U.S. PILOT - PHASE I - SHAKEDOWN TEST 
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EVALUATION OF WEIGHTED AGGREGATION PROOF DURE (1IAP)* 
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GENERAL OVERVIEW OF APPROACH 
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EVALUATION OF MACHINE FUNCTIONS. 
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PURPOSES OF BASELINE CORN AND SOYBEANS PROPORTION ESTIMATION PROCEDURE 
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FUNCTIONAL FLOW OF THE BASELINE CORN AND SOYBEANS PROCEDURE 
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CROP PROPORTION? 
SUMMER CROP RATIOS? 



QUESTIONS ADDRESSED BY THE EVALUATIONS - (CONTINUED) 
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SELECTED CHARACTERISTICS OF SEGMENTS OR BLOBS 
(CANDIDATES FOR INDEPENDENT VARIABLES) 
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Z of segment's pixels contained in interiors of "Big'' blobs. 
Average number of interior pixels per "Big* blob. 

Weighted average of "Big" blob purities for ... 

... Most prevalent crop type: C, S, or 0. 

... Most prevalent DFS crop group: SC, SG, P-V, N-V. 







CORN 


100 


CRITERIA FOR ASSIGNING GROUND-TRUTH CROP-TYPE LABELS 
TO "BIG" BLOBS ACCORDING TO PERCENTAGES OF INTERIOR PIXELS 



ORIGINAL PAGE IS 

OIF POOR QUALITY 


CRITERIA FOR ASSIGNING GROUND-TRUTH CROP-TYPE LABELS 
TO "BIG* BLOBS ACCORDING TO PERCENTAGES OF INTERIOR PIXELS 


ORIGINAL PAGE 13 
OF POOR QUALITY 
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GROUND TRUTH CROP TYPE PROPORTIONS OF SAMPLED BLOBS FROM SEVERAL SEGMENTS 

ERIM RESULTS - OCTOBER 7, 1980 


ORIGINAL PAGE IS 
OF POOR QUALITY 



39Vira3d NH03 


SOYBEANS PERCENTAGE 


SUMMARY OF THE ERIM STUDY RESULTS 


i5'3/\L PAGE *5 
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SOYBEANS PERCENTAGE 




n~v;rj«L p.'plT IS 

OF POOR QUALITY 


CRITERIA FOR DETERMINING WHETHER STAGE-TWO CROP-TYPE 
LABELS ASSIGNED TO "BIG" BLOBS BY ANALYSTS ARE "CORRECT" 

(NOTE: IF PERCENTAGES OF CORN AND SOYBEANS AND "OTHER" 

IN THE BLOB'S INTERIOR PIXELS CORRESPOND TO A POINT 
IN A REGION CONTAINING MORE THAN ONE OF THE 
SEVEN PERMISSIBLE LABELS, EACH OF THE LABELS 
IN THAT REGION SHOULD BE CONSIDERED AS 
"CORRECT" FOR THE blob) 
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30 40 50 60 70 
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53 



ORIGINAL PAGt IS 
OF POOR QUALITY 


CRITERIA FOR DETERMINING WHETHER STAGE-TWO CROP-TYPE 
LABELS ASSIGNED TO "BIG" BLOBS BY ANALYSTS ARE "CORRECT" 

(NOTE: WHERE MORE THAN ONE OF THE SEVEN PERMISSIBLE 

LABELS ARE SHOWN BELOW, EACH OF THE LABELS SHOWN 
SHOULD BE CONSIDERED "CORRECT" FOR A BLOB WHOSE INTERIOR 
PIXELS CONTAIN CORN, SOYBEANS, AND "OTHER" IN THE PERCENTAGES 
SHOWN) 
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TESTS AND EVALUATIONS OF THE BASELINE 
SEGMENT-LEVEL AREA PROPORTION ESTIMATION 
PROCEDURE AND ITS SUBCOMPONENTS 
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OVERALL PERFORMANCE OF THE BASELINE SEGMENT-LEVEL AREA PROPORTION ESTIMATION PROCEDURE 
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ACQUISITIONS THAT WERE AVAILABLE 



OVERALL PERFORMANCE OF THE BASELINE SEGMENT-LEVEL AREA PROPORTION ESTIMATION PROCEDURE 

(continued) 
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EVALUATION OF PERFORMANCE IMPROVEMENT OF BASELINE CORN/SOYBEAN PROPORTION ESTIMATION PROCEDURE 

OVER LAC1E TY PROPORTION ESTIMATION PROCEDURE 
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PS U = PROCEDURE X SEGMENT INTERACTION. 
ST ik m = ERROR. 



EVALUATION OF PERFORMANCE IMPROVEMENT OF BASELINE CORN/SOYBEAN PROPORTION ESTIMATION PROCEDURE 

OVER LAC IE TY PROPORTION ESTIMATION PROCEDURE 
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Methodology: anova for balanced incomplete block design. 


EVALUATION OF PERFORMANCE IMPROVEMENT OF BASELINE CORN/SOYBEAII PROPORTION ESTIMATION PROCEDURE 

OVER LACIE TY PROPORTION ESTIMATION PROCEDURE 
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ORIGINAL PAGE ® 


BASELINE CORN/SOYBEAN vs LAC IE TY CORN/SOYBEAN PROCEDURE EXPERIMENTAL PLAN 


Segment 

1 . 

2 . 

3. 

4 , 

5. 

6 , 

7. 

8 . 

9 . 

10 . 

11 . 

12 . 

13. 

14. 

15. 

16. 

17. 

18. 


LACIE TY 
Analyst Team 

A1 A2 A3 


x x 

x x 

x x 

x x 

x x 

x x 

x x 

x x 

x x 

x x 

x x 

X X 

X X 

X X 

X X 

X X 

X X 

X X 


C/S BASELINE 
Analyst Team 


B1 B2 B3 

x x 
x x 

x x 

x x 

x x 

x x 

x x 

x x 

x x 

x x 

x x 

x x 

x x 

x x 

X X 

X X 

X X 

X X 


Each segment will be processed by two of the three analyst teams 

USING BASELINE C-S PROCEDURE. 
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EVALUATION OF ADAPTABILITY OF BASELINE CORN/SOYBEAfi PROPORTION ESTIMATE PROCEDURE 

TO DIFFERENT CROP YEARS 

• OBJECTIVE 
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AS TYPE A 


EVALUATION OF ADAPTABILITY OF BASEL I TIE CORN/SOYBEAN PROPORTION ESTIMATION PROCEDURE 

TO DIFFERENT CROP YEARS 
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ORIGINAL PAGE IS 
OF POOR QUALITY 


EFFECT OF YEAR ON BASELINE CORN/SOYBEAN PROCEDURE EXPERIMENT PLAN 



TEAMS : A 

ANALYSTS: Al, A2 


B 

Bl, B2 


C 

Cl, C2 


1978 

1979 

Segment/ 

Analyst 

Analyst 

1. 

A1 

• — 1 
CO 

2. 

A1 

Cl' 

3, 

B2 

A1 

4. 

C2 

Al 

5, 

B2 

Cl 

6, 

C2 

Bl 

7. 

A2 

B2 

3. 

A2 

C2 

3. 

B1 

A2 

10. 

Cl 

A2 

11. 

B1 

Cl 

12. 

Cl 

B2 
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EVALUATION OF EFFECT OF FIELD SIZE ON LABELING ACCURACY AND PROPORTION ESTIMATION ERROR 
FOR BASELINE CORN/SOYBEAN PROPORTION ESTIMATION PROCEDURE 
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EVALUATION OF EFFECT OF FIELD SIZE ON LABELING ACCURACY AND PROPORTION ESTIMATION ERROR 
FOP BASELINE CORN/SOYBEAN PROPORTION ESTIMATION PROCEDURE 
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EVALUATION OF EFFECT OF FIELD Sl?£ ON LABELING ACCURACY AMD PROPORTION ESTIMATION ERROR 
FOR BASELINE CORN/SOYBEAN PROPORTION ESTIMATION PROCEDURE 


CO 

Z 

Ui 

F— 

Ui 


c 

< 


</> 

Ui 

z 








z 

o 





• 








< 














e. 

a. 





X 




• 





O 





F- 




F— 




UJ 

z 





3 

- 



CO 




_i 

o 





Z 

;r 



Ui 




OQ 






F- 

o 



z 




< 

CO 





1 




UI 

• 



> 

< 





a 

CO 



F- 

F— 



Z 






z 

CO 



Z 

CO 



UI 

a 





3 

UJ 



MM 

UJ 



Ui 

UI 





O 

z 




Z 



OQ 

-i 





z 

CD 



u. 

UI 



O 

UI 





CD 

UI 



o 

F- 




OQ 






z 




Z 



F- 

< 









a. 

MM 



< 

_) 





UI 




o 




Ui 






Nl 

z 



Z 

u. 



a 

CO 





M 

< 



CJ 

O 




CQ 





CO 

Ui 






• 

< 

O 






z 



Z 

Q. 


z 

-i 

-l 





a 

M 



O 

o 


o 


OQ 





_j 

_l 



u. 

z 


M 

o 






UI 

z 




o 


Ui 

F- 

U- 





M 

o 






Ui 


O 





u. 

z 



X 

Ui 


UI 

UJ 







>M* 



I- 

< 


Z 

M 

UJ 





UI 




3 



CD 

MM 

Nl 





CD 

z 



X 

a 


UI 

Ui 

MM 





< 

< 



F- 

U) 


Z 


CO 





z 

UI 



1 

..1 



a 






UI 

z 



a 

Ui 




UI 





> 

MM 



z 

aa 


Z 

UI 

CD 





< 

_J 



3 

< 


< 

MM 

< 



UI 






o 

-i 


UI 

u. 

z 



-l 


CO 

-1 



z 



z 


UI 


Ui 

OQ 


MM 

< 



cd 

Ui 


M 

u. 

> 


mJ 

< 



CJ 


-• 

w 

CQ 


_J 

o 

< 


CQ 

MM 


UI 

MM 


Ui 


O 


z 



UI 

< 

z 


-J 

I- 


z 

UJ 

-i 


o 

c. 




< 


CQ 

•n 


< 

M 

OQ 


z 

MM 

• • 

+ 

z 

> 


< 

MM 



MM 




X 

UI 


< 


z 

J* «• 

1- 


CO 

CO 

U. 



Ui 

-1 

X 

> 

F- 

o 

z 

< 


UI 


O 


z 

z 

CQ 

CQ 


z 

z 

< 

F— 


_J 

a 



< 

o 

< 


F- 

UI 

z 

> 

CO 


OQ 

_i 

UI 


UI 

MM 


+ 

Z 

Q 

UI 



a 

< 

UJ 

INI 


z 

F- 

z 


Ui 



F- 


UJ 

MM 

MM 

Ml 


MM 


< 

< 

Q 

UI 

X 

z 

• • 

3 

z 

u. 

Ui 


mJ 

□ 

> 


Z 

a. 

o 

UI 

> 

Z 

< 





UI 


II 

UJ 

UJ 

a 

a 

CD 

«— • 

> 

IU 

UJ 


-I 

z 

F- 


a. 

a 

z 

z 

O 

F- 


cd 

CD 

• • 

< 


z 

> 

UJ 

z 

< 

UJ 

-J 

Z 

F- 

< 

< 

>- 

CJ 

u. 

UI 


Q 

M 

z 

a. 

O 

O 

Z 

a 

oc 

CD 

MM 

o 

a 

• • 




UJ 

a 

CJ 

UJ 

UI 

UI 

o 

F“ 


Z 

-1 

II 

II 

II 

a 

o 


Q 

> 

> 

«J 

Ui 

z 

UI 

UJ 




z 

X 


Z 

< 

< 

o 

MM 

o 

a. 

a 

> 

X 

UI 

MM 

F— 

1 

UI 



a 

F- 

MM 

Ui 

O 





UI 


a. 



o 

< 

F- 

a 

i 





£ 

2 

UI 

• 

• 

X 

F- 

< 








o 

a 

< 

CQ 

F~ 

Ui 

Z 

i 

i 





I 

■M 

z 



UJ 











< 

z 

3 

UI 

mJ 1 

1 h* 

< 

UJ 

> 

a 

UI 



68 


Functions of Teams gf Two Analysts 
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For each segment processing 
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ANALYST FUNCTIONS (CONT.) 

FORf'«T OF CONFUSION MATRIX OF 
ANALYSTS’ DECISIONS VS. "CORRECT" 
DECISIONS FOR EACH SEASONAL CROP 
GROUP 
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Evaluation: 

For each seasonal crop group, plot the relationships, for each of its 

FIVE BIOWINDOWS, BETWEEN THE "DEVIATION* (AS THE DEPENDENT VARIABLE) 
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Analyst Functions (cont. 


original page is 
of POOR QUALITY 
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Selection of TPC acquisitions and assignment of TPC # s to DFS Strata. 

• Objectives: Describe the accuracy, and identify and assess factors which 

AFFECT THE ACCURACY, OF THE SEPARATION OF THE PIXELS IN EACH 
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PIXEL GROUND TRUTH l MILLS 


ORIGINAL PAGE IS 
OF POOR QUALITY 


FORMAT OF CONFUSION MATRIX OF GROUNO TRUTH CROP TYPES VS. TPC-OETERMINEC „u"- FUNCTION STRATA 


NUMBER OF PIXELS 
OF GRANO TOTAL 
t OF ROW TOTAL 
S OF COLUMN TOTAL 

CORN 

SOYBEANS 
ALL "OTHER" 
COTTON 
SUNFLOWERS 
SORGHUM 
WHEAT 
BARLEY 
RICE 
PASTURE 
GRASS 
TREES 
WATER 
URBAN 

ETC. 

COLUMN 
TOTALS 
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FORMAT OF CONFUSION MATRIX OF GROUNO TRUTH CROP GROUPS VS. TPC- DETERMINED DELTA FUNCTION STRATA 
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Evaluation: Plot relationships between selected entries in the confusion 

MATRICES (AS DEPENDENT VARIABLES) AND SELECTED SEGMENT OR BLOB CHARACTER I S~ 
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Final Selection of Corn/Soybean separation acquisitions., and placement of 

LINEAR DISCRIMINANTS ON SCATTER PLOTS. 
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II. Analyst Functions leading directly to labels for sampled "Big blobs. 

A. Final selection of acquisitions for blobbing, and assignment of "Stage One 
labels for sampled "Big" blobs. 

- Crop Group labels: "SC" (Summer Crop); "0" (Other). 
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ENTIFY AND ASSESS FACTORS WHICH AFFECT 
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For each segment processing, tabulate a "confusion matrix* for the 

PIXELS IN THE SAMPLED "BIG* BLOBS, SHOWING SPECIFIC GROUND-TRUTH 



CONTAINED IN SAMPLED "BlG* BLOBS. 


ORIGINAL PAGE 13 
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ORIGINAL PAGE IS 
OF POOR QUALITY 


ANALYST FUNCTIONS (CONT.) 
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Evaluation: Plot relationships between selected entries in the confusion 
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Assignment of "Stage Two 7 Labels to sampled '313*' Slobs (by individual analyst) 


ORIGINAL PAGE IS 
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Analyst Functions (cont. 
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PIXEL GROUND TRUTH lABELb 


ORIGINAL PAGE IS 
OF POOR QUALITY 


analyst functions (CONT.) 
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ANALYST FUNCTION (CONT.) 
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LABELING ACCURACY MATRIX: CROP-GROUP LEVEL 

(NUMBERS OF BLOBS. WITH JOINT- AND CONDITIONAL-PROBABILITIES) 
(HYPOTHETICAL EXAMPLE-NOT ACTUAL RESULTS OF ANY EXPERIMENT) 
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ANALYST FUNCTIONS (CuNT. ) 
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Analyst Functions (cont. 
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II. MACHINE CONTRIBUTIONS TO VARIANCE OF AREA PROPORTION ESTIMATES 
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USING THE POPULATIONS OF BLOBS FORMED DURING ALL ACTUAL PROCESSING OF SEGMENTS 
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DFS CROP 6RGUP PURITY 



PIKl GROUND TRUTH LABELS 
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using (all "big'Vall "smallVall) blobs in each segment which have the appropriate 
"most PREVALENI Cfi'-P* type", plot the following relationships for all segment 
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TRUE PR0°. OF TRUE PROP. OF TRUE PROP. OF 

CORN IN SEG. SOYBEANS IN SEG. "OTHER" IN SEG. 
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EVALUATION OF CROP DEVELOPMENT STAGE MODELS FOR CORN AND SOYBEANS 
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EVALUATION OF WEIGHTED AGGREGATION PROCEDURE (WAP) 
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ACQUISITION HISTORY EFFECTS 



SUMMARY OF RID* s RECEIVED 


An experiment overview and the general significance of the Individual 
evaluations Is needed In the Experiment Deslcn to support the technical 
evaluations presented. 

A prioritization of tests and evaluations Is requested by rank of Impor- 
tance to support upcoming experiments In the event that resources become 
no. available to perform all proposed evaluations. 

Evaluation criteria for the planned quantitative evaluations are needed. 

Qualification of the evaluation of machine bias contributions to the area 
proportion estimates Is neeaed. 

An evaluation of the impact of eliminating analyst team requirements on 
the procedure performance is needed. 

Operations plans and a defense of the experiment schedule within scheduled 
resources are needed to assure the feasibility of the experiment design. 

The ERIM/UCB corn/soybean consortium's role in the Pilot Experiment 
needs to be defined. 

Organization of the experiment Gesign such that "segment effects" can be 
clearly designated Is needed. 

A description of the data set in terms of agronomic features Is needed 
to aid post-processing stratifications. 

Evaluation of the procedure performance under an acquisition history 
representative of single satellite coverage Is requested. 

Evaluation of the effect of observed acquisition histories on the procedure 
performance Is needed. 
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Evaluation of the effect of crop calendar accuracy on the procedure 
performance Is needed. 

Evaluation of the procedure subcomponent accuracies on the procedure 
performance Is needed. 

Evaluation of the effect of film product quality on analyst performance 
Is needed. 
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RIO'S RECEIVED 


RIO NO. 


SOURCE 

1 

Experiment overview lacking 

T. Pendleton, NASA 

2 

Experiment overview lacking 

M. Metzler, ERIM 

3 

Prlortlzatlon of evaluations 

M. Metzler, ERIM 

4 

Power of quantitative evaluations 

T. Pendleton, NASA 

5 

Power of quantitative evaluations 

T. Pendleton, NASA 

6 

Analyst team requirements 

T. Pe-.dleton, NASA 

7 

Quantification of machine biases 

B. Thelen, ERIM 

8 

Analyst team requirements 

M. Metzler, ERIM 

9 

Operations plans and experiment schedule 

K. Baker, NASA 

10 

C/S consortium's role in the pilot experiment 

M. Metzler, ERIM 

11 

Interface definition - role of ERIM 

R. McKinney, ERIM 

12 

CHARLIE access software requirements 

K. Baker, NASA 

13 

Quantification of "Segment" effects 

M. Metzler, ERIM 

14 

Description of segment characteristics 

M. Metzler, ERIM 

15 

Acquisition history study 

M. Metzler, ERIM 

16 

Crop calendar accuracy 

M. Metzler, ERIM 

17 

Subcomponent accuracies and procedure 



performance 

K. Baker, NASA 

18 

Film product quality and analyst performance 

R. McKinney, NASA 

19 

Acquisition history and procedure performance 

R. McKinney, NASA 
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WIHtl 

27 Jan 

nil i cv mi oi jrosi nun 
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mums 


ill Ik t iiNSa 

SH2 

UMnMMH 

HH 



nuier 


POR of BL C/S Procedure 

IKtlftlM <tf 

The preliminary design of the experiment, as presented, appears to contain many 
worthwhile tasks, both statistical tests and parameter collection and display. 
However, the presented design lacks an overview of what questions are being 
addressed, what tasks address each question, and what type of result Is expected 
from each task. ("Result" Is meant to kiply "power of test", utility parameter 
collection, etc. ) 


IMPACT 1 9 MeQ»MtttO*r IQm *OT KCIMCO 

High probability that Important aspects of the 81 C/S procedures will not be 
evaluated adequately. 


QftiaiAATOA*? •tCOMMtnOAf ION 

Develop the experiment overview, have the overview reviewed by FCPF and appropriate 
personnel, and hold another PDR. 
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wam( 

IJOMATUM 


in, [['Iff 

T. Pendleton 
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IUUICT 

Key experiment design question 


•UCJMtriOft 99 

In description of tests, key questions or go^ls of tests were difficult to 
perceive. 


impact if MCONttCnOAr tow *»or *cctprco 

Value of tests may be lost due to lack of perceived focus, 


icamUMOArt o*» 


Experiment design questions should be clearly identified and listed according to 
Importance. Organization under major headings like: 1. analyst effects; 2. machine 

effects; 3. segment; 4. year; 5. sensor; and 6. procedure effects. 



disposition 


Q «**•««(■ *i?h tfoaincATio* 
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Scope of experiment 


A large number of tests have been proposed which address many aspects of the 
performance of the procedure. It Is unclear which tests are more Important 
or should be eliminated In case of a resource Impact. 


IM#*CT If MC9»i'«»0Af tOM *»OT aCCtfTtO 

Important tests may be eliminated. 
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Prioritize tests. 
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POR of BL C/S Procedure 
777e*i»n«* o* 

In some Instances, It appears not possible to estimate the power of the Individual 
tests or experiments prior to running the experiments but It will be possible to 
estimate the power after the experiment Is run. The experiment design does not 
provide for such analysis. 


I tt# *CT i 9 KCOMMCmOAr ton *0? acCIMIO 

Analysis may not take place because the appropriate data was not acquired during 
the experiment. 


0*1 ® IftAfO** 1 MCjMlNOAMON 

Plan, where required, analysis of experiment power. 
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Power of tests to be applied to BL C/s P 
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The experiment design does not appear to address the power of the tests to be 
conducted. 


IM#aCT if «CCO»MCNO*no« *or tCCCF'CO 


Inappropriate tests may be conducted or tests which will give no information 
will be conducted. 


o*iaiftAro**s Me4Mtt«»o*rio« 

Address the power of each of the tests to determine the significance level at 
which the affects will be detected, since the data set available for the experiment 
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PDR of BL C/S Procedure 


•IMIlpriKl If MIKIm 

Tht> BL procedure requires analyst teams for the purpose of Improving consistency. 
The trade-off Is efficiency. The experiment plan does not address this trade-off 
between consistency and efficiency. 


i» •icaMa«»g«r i«m iccirio 


A major factor In the use of remote sensing technology for commodity estimation 
Is being Ignored. 


ICOMUtNOAf H 


Design an experiment to address the trade-off between consistency and efficiency 
relating to analyst teams. 
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Biases need to b« summarized In a statistical type of setting. 


IMPACT 1 9 MCOweeCM A? to* *«T *CCIAM» 


Difficult to analyze biases and characterize them If this Is not done, especially 
If they are not large In magnitude. 


IC3mtt«0»ri on 


By subsetting segments, determine subset effects, 
biases, as well as variance. 


Finally, give average of all 
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Analyst Teams 





Reduction of analyst contract time of 33 percent would be gained by not using an 
analyst team. 


MR# RCV \f •ICOMleliriM Mf »CCUMI 

A more efficient team must be developed to support phase II pilot. 


9tt<l«s*rt«* t «tC4aaae(nCATl«N 


Consider Impact on experiment design to evaluate Impact of eliminating team on 
Procedure performance (especially consistency in Stage 1 labels). 
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t Throughput Rati 


No operations analysis of the segment arv.a estimation process was presented to 
Indicate whether or not the resources planned are sufficient to carry out 
the number of segment processings that are planned on the current schedule. 

The amounts of analyst and calendar time consumed In the shakedown so far suggest 
that the experiment will not be completed within the planned experiment size, 
schedule, and resources. 


iwicr 1 1 ««r tccirris 

We may get an unpleasant surprise part way through the experiment by discovering 
then that the experiment Is overscoped when we could have predicted It before 
we started. 


6«t cnuro* i MeaaMt<i«OAno« 


Perform a throughput study of the segment analysis process that will quantify 
the relation between experiment design, schedule and manpower for the ERIM 
procedure and the JSC feci 11 ties for carrying It out. Use results to advise 
management as to whether planned experiment can be completed on schedule and 
within budget. 
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ERIM's role In quality assurance has not been Identified. 


IMPACT 1 9 aCCOMMCNOAriOH not ACCIPTCO 


ERIM will be unable to confirm the results of the experiment. 


fl»l & tdirs * 1 5 MC3»M««0»TI «■ 


ERIM examined the analyst's decisions, etc. for subset (10%) of segments processed 
as well as independently processed some of the segments in totality. 
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The Interface between the U.S. C/S Pilot experiment and other organizations such 
as ERIM and LARS has not been defined nor acknowledged. 


i«#*cr \f •cc?MMC*o*riQft %ot kcimco 


The essential interfaces ,.'ay not occur for efficient project completion. 
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Define and formalize the interfaces between LEMSCO and other agencies necessary 
to complete the project. 
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Requirements for CHARLIE Access Software for Accuracy Assessment 


or ••oaiiM 

Many of the studies planned by accuracy assessment to evaluate various steps on the 
Area Estimation Procedure require results that are stored by the ERIM area estima- 
tion software In the segment analysis data base called CHARLIE. Thus, access 
routines to retrieve this data for use by accuracy assessment programs are needed. 
No written requirements exist that describe these needs, and ERIM, by their own 
statement, Joes not know exactly what Is needed. 


i f MCoimCNOAr iq» ior *ccimco 

Accuracy Assessment will either not have access to the segment analysis results 
they need or they will have to spend their own resources writing access routines 
for a data base whose internal structure they are not familiar with, since no 
program documentation has been delivered by ERIM yet. 


0*iaiM*ro*'t MCOiweatnOAT I ON 

Accuracy assessment whould document in writing their requirements for access 
routines that would return to their programs from CHARLIE the segment analysis 
results that they need. 


I f 

■■tfttXLTnTlMHl 



OtSMSinON 


O **»••*«• «tTM ■••mciritu 


!<•((■ I M l(Mt ettlMM 



JSC p«fa 1631 (It* Ott 74) 


135 













OR.'JifiAl. PAGE 13 
OF POOR QUALITY 


WIHIII 




26/27 January 81 


REVIEV ITEM OIJfOSITION 


uiimirmi i 


ifijfnvufi- 



M. Metzler ERIM 


iwjier 

Segment effects 


•iiuoriM tf ••••!.(« 

A cohesive design for uncovering segment effects was missing, though It appeared In 
several spots. The power of the experiment with segment. effects Is unclear. 


!M#ACT If MCOMMCMOtr 10 * *or ICCEMCO 


The power of segment- effect* analyses will not be understood. 


OftiaifiAro** s MCduwCHOAn o« 


Design the experiment In a manner that organizes all segment effects under a single 
umbrella. Use a model to explain this: (eg y * u + A + S + e, where A is acquisi- 
tion effect, and S Is signature effect). 
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Satellite Effect 
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A progamma ti cal ly Important question relates to the need of a 9- day versus a 
18-day coverage requirement for crop Inventory; there Is some potential In 
the 1978 data set to address this Issue. 


impact \f HCoftMeCftoortoN "0* acciamo 


Postpone, but possibly will need to plan as exploratory with the emphasis. 


OAiaifcAToa*! sccdwucftOArtdtt 


Process a subset of segments (resources permitting) with one and two sensors, and 
test hypothesis that two sensors result in better performance. 
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Value of crop calendar experiments. 
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Although accuracy of the crop calendar adjustment 1$ tested, the accuracy required 
for the procedure Is not addressed. 


impact IP «fCQm*|NO*rifl« "CT *CCtPMO 


It will not be known how Important (good) crop calendars are to the proportion 
estimates. 




Process subset of segments using various crop calendars (analyst historical, 
"correct") and evaluate proportion estimates produced. 
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The experiment design Includes many small studies of the performance of various 
Individual steps In the segment area estimation process. However, few plans 
appear to have been made to directly determine the effect of these steps on the 
quality of the final answer. 


impact if ttcoMttCftOftriON *of *cc(pmo 


The experiment may not produce any estimate of how much the overall accuracy of 
the procedure could be Improved nor any reliable indicators as to which steps in 
the procedure are most Important In its overall performance. 


OP I OlffAfOO* § MC3*«<i»OAr!0>» 


The experiment design should be modified to include more study of the effect that 
the accuracy with which the various intermediate steps are carried out has on the 
accuracy of the final result. 
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Establish Relationship Between Film Products Quality and Analyst Performance 


aticaiariaa «» ’Malta 

llght/dark contrast, textural distribution, and other physical qualities of the 
film products used by the analysts In labeling vary widely between acquisitions and 
sample segments. These qualities seriously affect the accuracy of analyst perfor- 
mance. This fact Is Ignored In the design of the many experimental studies pro- 
posed to study various aspects of procedure Involving analyst performance. The 
visual perception quality of film products should be considered. 


if *or •ccimio 

The influence of film product quality and its effect on analyst performance, 

If Ignored, may lead to erroneous conclusions regarding the Importance of other 
experimental variables. 


o*i«iN/ra«*i aic9HMiMO*f io« 

Attempt to scientifically establish the relative perceptual qualities of film 
products used In the proposed experimental studies that this influence can be 
rationally evaluated. 
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" , * 4<cr Establ1 S h relationship between number and distribution of sample segment 
acouisltlons and analyst performance 


Experimental studies proposed comparing sample segments and Involving analyst 
performance did not reflect consideration of the total number of acquisitions 
per sample segment, nor their distribution over the growing season. Differences 
between such sets seriously affect analyst performance. 


im»*ct ir .,r icciMtt 


Relationships between changes In experimental variables of Interest may be 
obscured by the effect of differences between sets of acquisitions for different 
sample segments. 


• t c *r i on 


Attempt to scientifically establish the differences between sets of acquisitions 
for various sample segments, rating them both for a combination of total number of 
Interest and their distribution over the growing season. 
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RESPONSE TO RID' s 


No. 

1, 2 EXPERIMENT OVERVIEW ANO GENERAL SIGNIFICANCE OF EVALUATIONS WERE 
LACKING 

1. A summary of the planned evaluations has been prepared and 

Included In the experiment plan which defines for each Individual 
evaluation: 

a. The general purpose and significance. 

b. The general approach to meeting the evaluation objectives. 

1. C tailed information has been added to each Individual evaluation 
ptan to define: 

a. The specific purpose, significance and key questions addressed 
by the evaluations. 

b. The detailed description of the evaluations Including the 
evaluation criteria. 


3 PRIORITIZATION OF EVALUATIONS 

The tests for the 1978, 1979, and 1980 data have been reviewed In 
terms of the project objectives: (1) Processing of the 12 1978 

segments In the year-to-year study have been delayed until all team 
objectivity and 1979 data processings are complete. (2) Evlauatlons 
of the results to be completed In FY1981 have been prioritized and 
presented to NASA management. 

Otherwise at this time, it appears that all proposed evaluations can 
be accomplished prior to completion of the pilot experiment in FY1982. 


4, 5 POWER OF THE QUANTITATIVE EVALUATIONS 

Evaluation criteria have been Incorporated Into the evaluation plans 
where a priori estimates of performance were available. For example, 
the power of the team objectivity and the procedure comparison 
analyses of variance have been estimated using empirical results from 
the LACIE TY exploratory experiment. 
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7 QUANTIFICATION OF MACHINE BIASES 

The biases Inherent In the proportion estimation calculations will 
be estimated as defined In the preliminary experiment design. In 
addition, evaluations of these biases have been quantified and Included 
In the evaluation plan. Simultaneous confidence limits will be 
completed and tha Importance of each machine bias and defined linear 
combination of biases will be addressed. 


6, 8 ANALYST TEAM REQUIREMENTS 

To Improve the processing efficiency of the baseline procedure In the 
pilot experiment over the shakedown test p-rforr-ance, the number of 
functions requiring full participation of an analyst team has been 
reduced (11). At the end of the processing of the 1978-79 data set, 
several analysts will have considerable experience. It Is felt that 
they will be able to judge what functions will require team decisions. 
Otherwise, considerable resources would be drawn from exlstinq evalua- 
tions to process a separate set of segments to quantify the difference 
In the Individual analyst and team performances . 


9 OPERATIONS PLANS AND DEFENSE OF EXPERIMENT SCHEDULE WITHIN SCHEDULED 

RESOURCES ARE NEEDED 

Proportion estimation has prepared and presented to project management 
an operations plan (17) to define whether the segment processings 
required to support the experiment design are within the scope of 
the pilot experiment. This operations plan was based upon the result 
of the shakedown test. The latest version of the plan based upon the 
early processings of the 1978-79 data set Is Included In the experi- 
ment plan. 


10, 11, ERIM/UCB CONSORTIUM'S ROLE(S) IN THE PILOT EXPERIMENT 
12 , 

Several roles supporting the pilot experiment have now been formally 
Identified. These Include: 

1. Image and CHARLIE library access software Is anticipated to be 
delivered by ERIM. Accuracy assessment has Identified for NASA 
the respective access software requirements to support the pilot 
experiment (8). 

2. For quality assurance, ERIM will verify the procedure Interpret- 
ation of the Lockneed personnel for a subset of segments In the 
team objectivity evaluation. If substantial problems with pro- 
cedure Interpretation are Identified, ERIM will present the results 
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to a configuration control team composed of the C/S Consortium, 
NASA, and Lockheed personnel. Upon review of the problem, this 
team will specify what action may be required. 

13 QUALIFICATION OF SEGMENT EFFECTS 

The current experiment design addresses the segment effects of field 
size, croo proportions, and crop ratios. These tests are uncontrolled.. 
However, they will capture whatever range the available segment data 
offers. The segment-level effects suggested In the RID (l.e., 
acquisition and signature effects) are also uncontrolled. Those 
effects are not easily quantifiable In a manner which can be modeled 
to an analysis of variance as suggested. 


14 DESCRIPTION OF SEGMENT CHARACTERISTICS 

The data set for the 1978-79 crop year evaluations has been strictly 
defined by the limits of the available data to support the experiment 
design for the team objectivity, procedure, and year-to-year evalua- 
tions. Subsequent to the PDR, accuracy assessment has summarized 
from UGTT ground truth Information the proportions of the major 
crops In the 1978 crop year da*.*. The additional segments are 
expected to be characterized when the necessary Information becomes 
available. These will be used to stratify the results In the course 
of the evaluations. 


15 ACQUISITION HISTORY STUDY 

To quantify the effect of acquisition history on the procedure perfor- 
mance would require drawing resources from the existing studies: 

1. Rankings of performance within the observed acquisition histories. 

2. Segment processings within the planned evaluations have been defined 
to be performed under either 10- or 9-day acquisition histories. 


Therefore, Insight Into the question can be gained by comparison of 
the results of the planned evaluations. 


16 CROP CALENDAR ACCURACY 

To quantify In a controlled experiment the effect of crop calendar 
accuracy on the overall procedure performance would be at present 
very costly. It Is preferable to defer such an analysis until the 
procedure Is automated to the level that the evaluation Is affordable. 
However, In an uncontrolled evaluation, the effect of correctly 
defined biowindows resulting from analyst Interp-etatlon of spectral 
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and crop calendar Information will be quantified as defined In the 
experiment plan. 


17 EFFECTS OF SUBCOMPONENT ACCURACIES ON THE PROCEDURE PERFORMANCE 

Quantitative evaluation of the subcomponent effects on the procedure 
performance would require standardization of the procedure preceding 
the subcomponent under assessment. The large number of processings 
that would be required to quantify the many combinations would be 
extremely costly. The experiment design was limited by the number 
of processings allowed within the scope of the experiment and the 
time demands of the procedure. Hence, the design was prepared to 
learn the most within the resources available, 60 processings. 

In addition, the experiment design addresses the question of sub- 
component effect upon procedure performance In uncontrolled evalua- 
tions. 


18 EFFECT OF FILM PRODUCT QUALITY ON ANALYST PERFORMANCE 

It Is not apparent that variables related to film quality can be 
operationally defined. Experiment design Is open to suggestions 
on how Image quality can be classified. Perhaps If a method Is 
conceived, this problem can be addressed In a later experiment 
design or techniques development. 


19 ACQUISITION HISTORY VS. PERFORMANCE: 

It appears difficult to define variables regarding the quality and 

the temporal distribution of available acquisitions whose relation- 
ships with other variables regarding procedure performance could be 

Interpreted. Such variables would have to take Into account: 

1. The "quality" of the Imagery for each available acquisition. 

2. The spectral biowindows of the crops of Interest for the growing 
seasons covered by the available acquisitions. 

3. The spectral separability of the crops of Interest on the imagery 
for each available acquisition. 

4. The criteria, the non-imagery data (crop calendars, etc.), and 
the procedures prescribed for estimating 1., 2., and 3. 

5. The criteria and the procedures prescribed for selecting acquisi- 
tions for processing from among the available ones. 
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6. The methods for "errors" in estimating 1., 2., and 3., and 
"errors" in selecting acquisitions for processing. 


It is not clear how the relative effects of these factors can be 
determined, for they seem to be interrelated and to require subjective 
judgment. 

It Is clear that the <r« Mty and the temporal distribution of the 
avallalbe acquisi . ,Jace an upper limit on the potential perfor- 
mance of any area proportion estimation procedure. But It Is not 
clear how to determine for a particular set of available acquisitions 
what Is the bast possible performance without testing It for every 
permissible combination of selected acquisitions. This Is so because 
It Is not clear how to determine In advance what is the "best possible" 
combination of selected acquisitions. For the same reason, it Is. 
not clear how to assess the relationships between available acqui- 
sitions and analyst, performance. 


147 


• OX OOVMNMNT MtNTMQ OWCI 


